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SUMMARY 

The  objective  of  this  research  was  to  investigate 
different  alternatives  for  pilot-computer  interaction  with  a 
simulated  expert  system.  Three  methods  of  advice  presenta¬ 
tion  were  investigated  —  automatic,  subject  requested,  and  a 
combination  of  the  two.  Also,  three  experience  levels  of  the 
subjects  were  examined  —  novice,  intermediate,  and  expert. 
The  dependent  variables  were  total  score  for  successful 
missions  completed  during  the  experimental  session  and 
average  time  to  complete  a  successful  mission.  The  analysis 
of  the  data  revealed  that  the  automatic  condition  was  the 
best  overall  method  for  advice  presentation  (p<.l)  and  the 
experience  level  of  the  subjects  was  not  a  significant 
factor. 


>  v 


€ 


s'- 

s'  's'- 

V-  •  * 
»v 

■J? 


Mi 


ACKNOWLEDGMENTS 

The  authors  would  like  to  thank  both  the  Frank  J. 
Seiler  Research  Laboratory  and  the  Aerospace  Medical 
Division's  Cockpit  Automation  Technology  project  who 
financially  supported  this  research.  The  efforts  of  SSGTs 
Bob  Stewart  and  Dave  White  in  the  modification  of  the 
simulator  were  also  greatly  appreciated.  -  v. 

TIC  TAB  H 

:  : j 

!  Jjvi  if  Jcation _ 


By — _ _ 

Distribution/ 

Availatility  Codes 
[Avail  nr<i/or 
Cist  1  Special 


1 


■ 


'.V.S 


,  *  *  1  « 
r  *.T  v’  « 


TABLE  OF  CONTENTS 


APPENDIX  C:  Questionnaires 


INTRODUCTION 


Although  a  fighter  mission  is  rather  complex,  the 
pilot's  duties  can  be  divided  into  five  broad  catagories: 
piloting  the  aircraft;  navigation  to  and  from  the  primary 
target;  weapon  delivery;  threat  avoidance  and  elimination; 


For  a  pilot  to  use  such  an  expert  system  effectively, 
the  interaction  with  the  system  should  be  designed  in  the 
most  efficient  manner.  This  paper  presents  initial  results 
from  a  research  effort  that  investigated  different  alterna¬ 
tives  for  advice  presentation.  The  three  alternatives 
investigated  were:  (1)  Automatic  -  the  mission  was  tracked 
and  advice  was  automatically  provided  to  enhance  performance; 
(2)  Request  -  the  mission  was  tracked  and  advice  was  provided 
only  when  queried  by  the  pilot;  and  (3)  Both  -  a  combination 
of  the  previous  two  methods.  Voice  synthesis  was  chosen  as 
the  mode  for  giving  the  advice  based  on  the  assumption  that 
future  cockpits  will  use  a  voice  input/output  system  as  one 
method  to  communicate  with  the  computer. 

Besides  the  presentation  style,  a  second  factor  to 
consider  in  designing  this  expert  system  is  the  varying 
levels  of  experience  of  the  pilots  who  will  use  it.  You 
would  expect  that  a  novice  pilot  would  be  helped  more  by  an 
expert  system.  As  a  pilot  gains  experience,  however,  their 
internal  model  of  the  mission  broadens  and  the  same  advice 
may  not  be  as  helpful  or  may  even  hinder  performance  if  the 
advice  conflicts  with  their  model.  Morris,  Rouse,  and  Frey 
(1984)  hypothesized  that  subjects  would  rely  more  heavily  on 
a  decision  aiding  system  when  their  perceived  performance  is 
poor  (i.e.,  inexperienced  subjects)  compared  to  when  their 
perceived  performance  is  good. 


METHOD 


Subjects 

Twenty-nine  male  USAF  Academy  cadets  from  all  four 
classes  served  as  volunteer  subjects  in  this  experiment. 
Subjects  were  initially  screened  so  that  only  those  with 
prior  experience  in  playing  F-15  Strike  Eagle  were  used. 
However,  ten  cadets  who  had  not  played  this  game  were  used  as 
novices  to  balance  the  sample  sizes  for  the  'experience' 
variable.  Also,  the  data  for  four  other  subjects  was  elimin¬ 
ated  from  the  analysis  because  they  failed  to  score  any 
points  during  at  least  one  of  their  sessions. 

Apparatus 

Simulation.  The  fighter  mission  was  simulated  using  the 
comercially  available  F-15  Strike  Eagle  video  game  developed 
by  MicroProse  Software  and  was  played  on  a  Kaypro  16  computer 
with  a  color  monitor.  This  video  game  simulates  several 
air-to-ground  mission  scenarios  with  varying  levels  of 
difficulty.  The  game  also  has  a  variety  of  threats  (i.e., 
enemy  aircraft  and  heat  seeking  and  radar  guided  missiles)  to 
deal  with  while  flying  the  missions.  There  are  a  total  of 
seven  missions  and  four  levels  of  difficulty  —  arcade, 
rookie,  pilot  and  ace.  In  addition,  each  successive  mission 
is  more  demanding  at  each  level  of  difficulty. 

To  present  the  necessary  information  for  the  missions, 
the  game  uses  four  primary  displays:  a  Head  Up  Display  (HUD), 
tactical  situation  display,  radar  electronic  warfare  display. 


and  a  pictorial  stores  display  (Figure  1).  The  offensive  and 
defensive  weapons  include  Electronic  Counter  Measures  (ECM) , 
flares,  dumb  bombs,  and  short  and  medium  range  missiles. 
Lastly,  systems  information  is  provided  for  fuel,  heading, 
airspeed,  and  altitude.  Therefore,  with  the  game's  very 
solid  aerodynamic  model  and  demands  placed  on  the  human 
information  processing  resources,  this  game  realistically 
approximates  the  fighter  cockpit  environment  quite  well. 

To  further  enhance  the  game's  realism,  it  was  played  in 
a  modified  Air  Force  T-38  simulator  that  contained  three 
color  Cathode  Ray  Tubes  (CRTs).  The  video  game  was  presented 
on  the  CRT  located  in  the  HUD  position.  The  subjects 
controlled  the  game  through  a  joystick  and  computer  keyboard 
that  had  also  been  mounted  in  the  cockpit.  In  addition,  four 
lighted  push  button  switches  were  mounted  on  the  forward  pos¬ 
ition  of  the  left  side  panel  for  the  subjects  to  request 
advice. 

Expert  system.  The  expert  system  advice  was  simulated 
by  an  experimenter  using  a  Texas  Instruments  Business  Pro 
computer  equipped  with  a  TI  Speech  II  board  allowing  text- 
to-speech  voice  synthesis.  The  expert  system  advice  was 
stored  in  the  computer  in  a  series  of  text  phrases  and  was 
selected  by  the  experimenter  from  a  menu.  An  example  of  one 
of  the  phrases  is:  "Warningl  Heat  seeking  missile,  deploy 
flare."  All  together  there  were  26  different  advice  prompts 


that  could  be  given  to  the  subject.  (See  Appendix  A  for  a 
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complete  listing  of  the  phrases  used.)  The  expert  system  was 
then  simulated  by  having  the  experimenter  track  the  mission 
on  a  separate  CRT  and  select  the  appropriate  advice,  depend¬ 
ing  on  the  experimental  condition,  to  be  presented  to  the 
subject.  The  subjects  were  presented  an  average  of  80  advice 
prompts  in  the  automatic  condition  during  an  experimental 
session.  Communications  among  the  expert  system,  subject, 
and  experimenter  occurred  over  an  intercom  system  through  a 
microphone  and  headset. 

Experimental  Design 

The  experimental  paradigm  used  for  this  study  was  a  four 
by  three  mixed  factor,  repeated  measures  design.  The  within- 
subjects  design  involved  the  expert  system  presentation  style 
and  had  four  levels:  automatic,  request,  both,  and  a  control 
where  no  advice  was  given.  To  eliminate  possible  learning 
effects  from  confounding  the  data,  the  order  of  presentation 
of  the  treatments  to  each  subject  was  counterbalanced  using  a 
balanced  latin  square  so  that  each  condition  was  preceded  and 
followed  equally  often  by  each  of  the  other  conditions. 

A  between  subjects  design  was  used  for  the  experience 
level  of  the  subject  and  had  three  levels:  novice,  inter¬ 
mediate,  and  expert.  Subjects  were  grouped  by  experience 
level  based  on  the  performance  of  a  pre-test  with  the  game. 
(See  the  Pre-Test  section  of  this  report  for  further  de¬ 
tails.)  All  subjects  completed  at  least  two  missions 


successfully  prior  to  the  experiment. 
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Dependent  Variables 

The  dependent  variables  were  the  total  score  for  the 
successfully  completed  missions  during  each  experimental 
session  and  the  average  time  to  complete  these  missions.  For 
a  subject  to  successfully  complete  a  mission,  they  were 
required  to  destroy  the  primary  target  and  safely  return  to 
home  base. 

Procedures 

Pre-test .  Prior  to  participating  in  the  experiment,  all 
subjects  were  pre-tested  on  the  video  game.  Each  subject 
flew  the  two  easiest  missions  at  each  level  of  difficulty, 
starting  at  the  rookie  level,  until  they  completed  the  two 
missions  at  the  ace  level  or  until  they  had  flown  a  total  of 
two  unsuccessful  missions.  Subjects  were  then  categorized  by 
experience  based  on  the  highest  level  of  difficulty  in  which 
two  missions  were  flown  successfully.  Beginners  completed 
two  missions  successfully  at  the  rookie  level,  intermediates 
completed  two  at  the  pilot  level,  and  experts  completed  at 
least  one  at  the  ace  level.  The  pre-test  was  administered 
approximately  one  month  prior  to  participation  in  the 
experiment  for  most  subjects.  The  subjects  were  also  told 
not  to  play  the  video  game  again  prior  to  their  participation 
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in  the  data  collection  portion  of  the  study  and  all  had 
compl ied . 

Experimental  sessions.  Subjects  flew  the  simulation  for 
forty  minutes  under  each  condition  on  four  different  days. 
Each  subject  flew  each  condition  at  approximately  the  same 
hour  of  the  day  and  had  at  least  one  rest  day  between 
experimental  sessions. 

Subject  briefing.  Prior  to  the  session,  the  subjects 
were  told  which  condition  they  would  see  that  day.  The 
experimenter  would  then  use  the  computer  and  text-to-speech 
synthesis  to  play  the  experimental  instructions.  (See 
Appendix  B  for  an  example  of  the  instructions.)  The  subjects 
were  then  reminded  of  the  condition  they  would  be  flying 
under  and  that  their  mission  objective  was  to  fly  directly  to 
their  primary  target,  attack  the  target,  and  return  to  home 
base  as  quickly  as  possible;  they  were  only  to  attack 
threats  that  endangered  the  success  of  the  mission. 

For  the  request  and  both  conditions,  the  subjects  were 
also  briefed  on  the  operation  of  the  four  switches  in  the 
cockpit.  There  was  one  switch  each  to  request  advice  on:  1) 
navigation,  2)  offensive  and  defensive  weapons,  3)  aircraft 
systems,  and  4)  a  switch  labeled  with  a  question  mark  to  be 
used  by  the  subject  when  they  did  not  know  what  specific  type 
of  advice  to  request. 


Data  collection.  Each  subject  started  each  session 
flying  mission  number  four  at  the  rookie  level.  If  the 
mission  was  flown  successfully  the  subject  then  flew  mission 
number  six  at  the  same  level.  However,  if  the  mission  was 
not  successful  (i.e.,  crashed,  shot  down,  or  failed  to 
destroy  the  primary  target),  the  subject  repeated  the  same 
mission  until  it  was  completed  successfully.  If  the  subject 
completed  these  two  missions  successfully  they  were  then 
advanced  to  the  pilot  level  of  difficulty  and  again  flew 
missions  four  and  six.  If  the  missions  were  successful 
again,  the  subject  advanced  to  the  ace  level.  Subjects 
advanced  in  this  manner  as  far  as  they  could  during  the  forty 
minute  session.  No  subject  was  able  to  complete  the  total  of 
six  possible  missions  during  the  session.  Upon  completion  of 
each  successful  mission  the  total  score  and  time  to  complete 
the  mission  were  recorded.  Following  each  session  the 
subjects  were  administered  a  questionnaire  to  collect 
subjective  data  for  that  condition  and  a  final  questionnaire 
was  given  following  all  four  treatments.  (See  Appendix  C  for 
examples  of  the  questionnaires.) 

RESULTS 

The  data  were  analyzed  using  the  MANOVA  for  repeated 
measures  procedure  in  the  SPSS/PC+  statistical  package  for 
microcomputers  (Norusis,  1986).  The  results  of  this  analysis 
revealed  only  one  significant  effect  which  was  the  main 
effect  of  the  advice  presentation  condition  (F=1.81,  p*.l; 
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Table  1).  In  addition  the  univariate  results  revealed  that 
only  the  total  score  contributed  significantly  to  this  result 
(F-2.63,  p*.056;  Table  2).  Figure  2  shows  a  plot  of  the  mean 
total  score  for  each  of  the  experimental  conditions. 
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Table  1 

Pillais  Summary  Table 
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Source 

Value 

Approx.  F 

Hypoth.  df 

Error  df 
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Experience 

.38360 
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(Within) 
Condition 
Condition  X 
Experience 

.13023 

.13909 

1.81089 

.97168 
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Table  2 

Univariate  Test  Results 


Variable 

df 

SS 

MS 

F 

P 

Score 

3 

33194711.0 

1106-4903.7 

2.62812 

.056 

Error 

78 

328395318.0 

4210196.4 

Avg.  Time 

3 

9.61820 

3.20607 

1.02602 

.386 

Error 

78 

243.73165 

3.12476 
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AUTOMATIC  REQUEST  BOTH  CONTROL 

CONDITION 

Figure  2.  Mean  Score  by  Condition. 

A  Duncan  post-hoc  test  was  then  used  to  compare  the 
means,  and  the  results  showed  the  automatic  condition  was 
significantly  different  from  the  request  and  control  condi¬ 
tions  (p<.l)  but  not  the  both  condition  (Table  3). 


Table  3 

Results  of  the  Duncan  Test  For  Differences 
Between  Means 


Request 

Both 

Control 

Automatic 

1310* 

541 

991* 

Request 

769 

319 

Both 

450 

The  number  of  requests  for  advice  made  by  each  subject 
in  the  request  condition  was  also  analyzed  by  experience 
level  but  no  significant  effect  was  found  (F<1).  All 
subjects  averaged  24.47  requests  per  session. 

Questionnaire  Results 

The  results  of  the  questionnaire  were  analyzed  using  a 
Chi-Square  goodness  of  fit  test  for  a  uniform  distribution 
(See  Appendix  C)  .  Several  interesting  findings  emerged  from 
this  analysis.  First,  26  subjects  agreed  the  advice  was 
essential  or  helped  quite  a  bit  in  the  performance  of  the 
missions  (X2*50.2,  p<.005/df»5) .  Second,  17  subjects  ranked 
the  automatic  condition  as  the  best  overall  method  for  advice 
presentation  (X2  =  9.44,  p< . 01/df =2 )  .  Finally,  all  the 
subjects  felt  that  threat  management  was  the  area  where  the 
advice  was  most  helpful  (X2»88.26,  p< . 005/df*3 ) . 

DISCUSSION 

Based  on  the  analysis  above,  several  points  can  be  made. 
First,  from  both  the  objective  and  subjective  data  it  is 
apparent  that  the  presentation  of  advice  did  improve  mission 
performance  and  that  the  automatic  condition  was  the  best 
overall  method  for  advice  presentation.  In  fact,  one  subject 
noted  in  his  final  comments  "I  was  really  surprised  at  how 
much  worse  my  performance  was  with  no  advice  as  compared  to 
my  performance  under  the  advice  condition.  I  think  the  voice 
talking  gave  you  a  certain  level  of  confidence  as  well  as 
pertinent  information."  In  today's  complex  fighter  cockpit 


environment,  the  pilot  does  not  have  the  attentional  resour¬ 
ces  available  to  monitor  every  aspect  of  the  mission.  By 
providing  the  automatic  advice  to  the  subject,  the  subject 


was  warned  of  critical  events  {e.g.,  threat  warnings)  that 
were  important  to  the  success  of  the  mission.  Therefore, 
information  processing  resources  only  had  to  be  diverted  to 
other  tasks  when  critical  developments  occurred  that  mandated 
attention. 

Second,  the  fact  that  the  automatic  condition  was 
better  than  the  request  condition  indicated  that  the  addi¬ 
tional  resources  required  to  request  advice  were  great  enough 
to  decrease  overall  mission  performance.  To  request  advice 
in  this  experiment,  the  subject  had  to  divert  attention  away 
from  the  primary  task  of  flying  the  mission  and  press  one  of 
the  switches  in  the  cockpit.  It  would  appear  that  in  a 
situation  where  the  pilot  is  already  extremely  taxed,  any 
additional  resources  allocated  for  other  tasks  can  cause 
performance  to  deteriorate.  Therefore,  an  expert  system 
should  keep  additional  requirements  for  information  proces¬ 
sing  resources  to  a  minimum  if  possible.  As  an  example,  an 
improvement  over  the  present-  configuration  might  have 
included  a  voice  activated  system. 

Third,  the  results  failed  to  show  a  significant  effect 
for  the  experience  level  of  the  subject.  It  was  expected  the 
advice  would  be  less  helpful  for  the  more  experienced 
subjects  since  they  had  already  built  an  internal  model  of 
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how  to  fly  the  missions  and  the  advice  might  have  interrupted 
their  thoughts  or  even  conflicted  with  their  plans.  This  was 
not  the  case  and  it  appears  that  even  expert  subjects  can 
benefit  from  expert  system  advice.  This  again  is  probably 
because  information  processing  requirements  were  high  even 
for  the  experts  and  the  expert  system  was  able  to  alert  them 
to  significant  events  they  may  not  have  detected. 

Finally,  no  differences  were  found  among  the  experience 
levels  of  the  subjects  as  to  how  much  advice  was  requested  in 
the  request  condition.  As  was  said  earlier,  Morris,  et.  al., 
(1984)  hypothesized  that  the  more  a  subject  perceived  their 
performance  to  be  poor,  the  more  they  would  rely  on  a 
computer  decision  aid.  In  this  study,  it  was  expected  that 
novice  subjects  would  perceive  their  performance  as  sub¬ 
standard  and  rely  more  heavily  on  the  advice  by  making  more 
requests  than  the  experts.  This  did  not  happen  and  it  seems 
that  the  novice  and  expert  subjects  perceived  their  perfor¬ 
mance  on  the  same  level  and  both  relied  on  the  advice 
equally.  However,  one  subject  did  state  in  his  final 
comments  that  "As  I  got  better  the  computer  advice  became 
less  helpful." 

One  final  point  that  needs  to  be  made  is  that  with  an 
expert  system  that  is  able  to  monitor  the  progress  of  a 
mission  and  provide  advice  to  the  pilot,  it  might  be  used  to 
further  reduce  the  pilot's  workload  and  improve  mission 
performance  if  it  could  actually  perform  some  of  the  mis- 
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sion's  tasks  for  the  pilot.  This  is  the  logical  next  step 
for  expert  systems  and  future  research  is  being  planned  to 
investigate  issues  dealing  with  how  the  pilot  would  interact 


with  such  a  system. 

CONCLUSIONS 

It  is  obvious  that  for  an  on-board  expert  system  to 
efficiently  contribute  to  the  performance  of  a  fighter 
mission,  the  pilot-computer  interface  must  be  understood  and 
specified.  This  research  supports  the  concepts  that  expert 
system  advice  should  be  presented  automatically  and  that 
additional  information  processing  requirements  due  to  the 
expert  system  should  be  kept  to  a  minimum. 
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APPENDIX  A 


Advice  Prompts  Used 

WARNING,  RADAR  GUIDED  MISSILE,  ACTIVATE  ECM. 

WARNING,  HEAT  SEEKING  MISSILE,  DEPLOY  FLARE. 

WARNING,  PULL  UP,  ALTITUDE  DANGEROUSLY  LOW. 

CAUTION,  ALTITUDE  TOO  HIGH  FOR  BOMB  DROP,  DESEND  TO  2000 
FEET. 

WARNING,  AIRSPEED  DANGEROUSLY  LOW,  INCREASE  THROTTLE. 

WARNING,  AIRSPEED  DANGEROUSLY  HIGH,  DECREASE  THROTTLE. 

WARNING,  ENEMY  AIRCRAFT,  FIRE  A  SHORT  RANGE  MISSILE. 

WARNING,  ENEMY  AIRCRAFT,  AIM  AT  THE  AIRCRAFT  AND  FIRE  A 
MEDIUM  RANGE  MISSILE. 

WARNING,  LOW  FUEL,  RETURN  TO  BASE. 

ADVISORY,  FOR  EFFICIENT  FUEL  USAGE,  REDUCE  THROTTLE  TO 
95  PERCENT. 

ADVISORY,  YOU  CAN  JETTISON  YOUR  DROP  TANKS. 

ADVISORY,  FOR  NAVIGATION  TO  THE  TARGET,  DESIGNATE  IT 
WITH  THE  NAVIGATION  CURSOR. 

ADVISORY,  FOR  NAVIGATION  TO  HOME  BASE,  DESIGNATE  IT  WITH 
THE  NAVIGATION  CURSOR. 

ADVISORY,  APPROACHING  PRIMARY  TARGET,  ARM  A  BOMB. 

ADVISORY,  FOR  EFFICIENT  FUEL  USAGE,  DISENGAGE  AFTER¬ 
BURNER. 

ADVISORY,  TO  INCREASE  AIRSPEED,  ENGAGE  AFTERBURNER. 

ADVISORY,  TO  LOOSE  ALTITUDE  FAST,  REDUCE  POWER  TO  55 
PERCENT,  AND  ACTIVATE  SPEED  BRAKES. 

ADVISORY,  YOU  HAVE  OVERFLOWN  THE  TARGET,  FLY  STRAIGHT 
FOR  A  FEW  SECONDS,  DO  A  LOOP,  AND  GET  BACK  ON  COURSE. 

CAUTION,  YOU  ARE  OUT  OF  MISSILES,  TO  DEFEAT  AIR 
THREATS  YOU  MUST  OUT  MANEUVER  THEM  OR  USE  GUNS. 


YOU  ARE  OUT  OF  FLARES,  TO  DEFEAT  HEAT  SEEKING  MISSILES 
YOU  MUST  OUT  MANEUVER  THEM. 

CAUTION,  YOU  ARE  OUT  OF  ECM,  TO  DEFEAT  RADAR  GUIDED 
MISSILES  YOU  MUST  OUT  MANEUVER  THEM. 

ADVISORY,  FOR  AN  EASIER  BOMB  RELEASE,  DECREASE  THROTTLE 
TO  55  PERCENT. 

ADVISORY,  YOU  ARE  OFF  COURSE. 

CAUTION,  ALTITUDE  TOO  HIGH  FOR  RETURN  TO  BASE,  DECREASE 
TO  3000  FEET. 

WARNING,  YOU  ARE  RUNNING  OUT  OF  FUEL,  CLIMB  AS  HIGH  AS 
YOU  CAN,  AND  DO  A  POWER-OFF  GLIDE  INTO  HOME  BASE. 

CAUTION,  YOU  SHOULD  DEACTIVATE  YOUR  SPEED  BRAKES  BY 
PRESSING  A  THROTTLE  SETTING. 


APPENDIX  B 


Instructions  to  the  Subjects 


HELLO.  I  AM  YOUR  ON-BOARD  INTELLIGENT  COMPUTER.  MY  PURPOSE, 
IS  TO  HELP  YOU  PLY  YOUR  MISSION  SUCCESSFULLY.  DURING  YOUR 
MISSIONS,  I  WILL  PROVIDE  YOU  WITH  ADVICE,  AUTOMATICALLY,  ON 
WAYS  TO  IMPROVE  MISSION  PERFORMANCE.  IF  YOU  LISTEN  TO  MY 
ADVICE,  YOU  WILL  BE  MORE  LIKELY  TO  COMPLETE  YOUR  MISSION 
SUCCESSFULLY.  HOWEVER,  IF  YOU  DO  NOT  LIKE  THE  ADVICE  I  GIVE 
YOU,  YOU  MAY  CHOOSE  TO  IGNORE  IT.  YOUR  MISSION  TODAY,  WILL 
BE  TO  ATTACK  THE  PRIMARY  TARGET  AS  INDICATED  ON  THE  TACTICAL 
SITUATION  DISPLAY.  SOMETIMES,  YOU  MAY  HAVE  TO  RETURN  TO 
BASE,  TO  REFUEL,  BEFORE  COMPLETING  YOUR  MISSION.  OUR  ATTACK 
ORDERS  ARE  TO  FLY  DIRECTLY  TO  THE  TARGET,  ATTACK  THE  TARGET, 
AND  RETURN  DIRECTLY  TO  HOME  BASE.  ONLY  ATTACK  THE  THREATS 
THAT  ENDANGER  THE  SUCCESS  OF  THE  MISSION.  IF  YOU  HAVE  ANY 
QUESTIONS,  ASK  THE  EXPERIMENTER  AT  THIS  TIME. 


APPENDIX  C 


POST  FLIGHT  QUESTIONNAIRE 
AUTOMATIC  ADVICE  CONDITION 

SUBJECT  NO.: _ 

Circle  the  appropriate  answer: 

1.  Did  you  find  the  computer  advice  helpful  in  the  performance  of  the 

mission?  _ 

X  =20.8;  p<.0001/df=4 

Not  at  All  Very  Little  Somewhat  Quite  a  Bit  Essential 

0  4  9  15  4 

2.  How  easy  was  it  to  understand  the  advice  that  was  given  to  you? 

X  =41.8;  p<.0001/df=4 

Very  Difficult  Difficult  Somewhat  Difficult  Easy  Very  Easy 

0  0  2  18  12 

3.  Was  the  timing  of  the  computer  advice  appropriate? 

X  =35.2;  P<.0001/df=4 

Unacceptable  Bad  Satisfactory  Good  Optimum 

03  15  14  0 

4.  Did  you  find  the  computer  advice  to  be  a  distraction? 

X  =58.5;  p<.0001/df=4  I 

Not  as  All  Very  Little  Somewhat  Quite  a  Bit  Always 

4  24  3  10 

5.  Rank  order  the  following  as  to  which  category  of  advice  was  most 
helpful  (l=most  helpful;  4=least  helpful; 

1  Threat  Management  (ECM,  Flares,  and  Missiles) 

4  Navigation  (To  the  target  and  home  base) 

2  Weapons  Management  (Bombimg  the  primary  target) 

3  Systems  Management  (Fuel,  Airspeed,  Altitude,  ect. ) 


6.  What  was  the  easiest  and  most  difficult  aspects  of  the  missions 
for  you,  and  did  the  advice  help  in  these  aspects? 

Easiest:  Navigation  (92%) 

Most  Difficult:  Threat  Management  (96%) 
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POST  PLIGHT  QUESTIONNAIRE 
REQUEST  ADVICE  CONDITION 

SUBJECT  NO. s _ 

Circle  the  appropriate  answer: 

1.  Did  you  find  the  computer  advice  helpful  in  the  performance  of  the 

mission?  2 

X  =10.5;  p=,033/df=4 

Not  at  All  Very  Little  Somewhat  Quite  a  Bit  Essential 

3  6  13  7  3 

2.  How  easy  was  it  to  understand  the  advice  that  was  given  to  you? 

X  =43.7;  p<.0001/df=4 

Very  Difficult  Difficult  Somewhat  Difficult  Easy  Very  Easy 
0  0  l  19  10 

3.  When  requested,  Was  the  timinq  of  the  computer  advice  appropriate? 

X  =26.1;  p<.0001/df=4 

Unacceptable  Bad  Satisfactory  Good  Optimum 

0  5  15  11  1 

4.  How  often  did  you  use  the  computer  advice? 

X  =14.5;  p=.006/df=4 

Not  as  All  Very  Little  Somewhat  Quite  a  Bit  Always 

2  8  5  14  3 

5.  Rank  order  the  following  as  to  which  category  of  advice  was  most 
helpful  (l=most  helpful;  4=least  helpful;  0=not  used): 

1  Threat  Management  (ECM,  Flares,  and  Missiles) 

3  Navigation  (To  the  target  and  home  base) 

2  Weapons  Management  (Bombimg  the  primary  target) 

4  Systems  Management  (Fuel,  Airspeed,  Altitude,  ect. ) 


6.  What  was  the  easiest  and  most  difficult  aspects  of  the  missions 
for  you,  and  did  the  advice  help  in  these  aspects? 

Easiest:  Navigation  (97%) 

Most  Difficult:  Threat  Management ( 100%) 
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POST  PLIGHT  QUESTIONNAIRE 
AUTOMATIC  AND  REQUEST  ADVICE  CONDITION 


SUBJECT  NO.  : _ 

Circle  the  appropriate  answer: 


1.  Did  you  find  the  computer  advice  helpful  in  the  performance  of  the 
mission?  , 

X  =34.6;  p<.0001/df=4 

Not  at  All  Very  Little  Somewhat  Quite  a  Bit  Essential 

o  0  11  17  4 

2.  How  easy  was  it  to  understand  the  advice  that  was  given  to  you? 

2 

Very  nitiltfl*'  4  Somewhat  Difficult  Easy  Very  Easy 

o  0  3  16  13 


3.  Was  the  timing  of  the  automatic  computer  advice  appropriate? 
X=24.6;  p<.0001/df=4 

Unacceptable  Bad  Satisfactory  Good  Optimum 


0  6  15  10  1 

4.  Did  you  find  the  automatic  computer  advice  to  be  a  distraction? 
X2=39.2;  p<0001/df=4 

Not  as  All  Very  Little  Somewhat  Quite  a  Bit  Always 

4  22  5  1  0 

5'.  How  often  did  you  request  advice? 

X2=5.8;  p=. 214/df=4 

Not  at  All  Very  Little  Some  Quite  a  Bit  Always 

3  10  9  6  4 

6.  Did  you  find  the  auotmatic  or  requested  advcie  to  be  most  helpful? 

X2=14.2;  p<.0001/df=l 

Automatic  Requested 

26  5 

7.  Rank  order  the  following  as  to  which  category  of  advice  was  most 
helpful  (l=most  helpful;  4=least  helpful 


1  Threat  Management  (ECM,  Flares,  and  Missiles) 

4  Navigation  (To  the  target  and  home  base) 

2  Weapons  Management  (Bombimg  the  primary  target) 

3  Systems  Management  (Fuel,  Airspeed,  Altitude,  ect. ) 

8.  What  was  the  easiest  and  most  difficult  aspects  of  the  missions 
for  you,  and  did  the  auotmatic  or  requested  advice  help  in  these 
aspects? 

Easiest:  Navigation  (90%) 

Most  Difficult:  Threat  Management  (98%) 
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PINAL  QUESTIONNAIRE 


SUBJECT  NO.: _ 

Circle  the  appropriate  answer: 

1.  Did  you  find  the  computer  advice  helpful  in  the  performance  of  the 

missions?  2 

X  *50.2;  p<.0001/df=4 

Not  at  All  Very  Little  Somewhat  Quite  a  Bit  Essential 

0  15  22  4 

2.  How  easy„was  it  to  understand  the  advice  that  was  given  to  you? 

X  *52.7;  p<.0001/df=4 

Very  Difficult  Difficult  Somewhat  Difficult  Easy  Very  Easy 

0  01  21  10 

3.  Rank  order  the  following  as  to  which  category  of  advice  was  most 
helpful  (l=most  helpful;  4=least  helpful; 

1  Threat  Management  ( ECM,  Flares,  and  Missiles) 

4  Navigation  (To  the  target  and  home  base) 

2  Weapons  Management  (Bombimg  the  primary  target) 

3  Systems  Management  (Fuel,  Airspeed,  Altitude,  ect. ) 

4.  Rank  order  the  methods  for  advice  presentation  as  to  which  method 
of  presentation  was  most  helpful  (l=most  helpful;  3=least  helpful; 


1  _  Automatic 

3 _  Request  Only 

2  _  Both  Automatic  and  Request 
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5.  What  was  the  easiest  and  most  difficult  aspects  of  the  missions 
for  you,  and  did  the  auotmatic  or  requested  advice  help  in  these 
aspects? 
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Easiest:  Navigation  (92%) 

Most  Difficult:  Threat  Management  (94%) 


